Remote Direct Memory Access (RDMA) provides read and write service directly between two applications and enables data transfers directly to application buffers without intermediate data copies. RDMA enables memory data transfers to bypass operating systems and kernels.
There are currently several protocols to transfer RDMA packets over networks. Examples of RDMA transfer protocols are RDMA over Converged Ethernet (RoCE) and Internet Wide Area RDMA Protocol (iWARP). RoCE is an Open Systems Interconnection (OSI) model Layer 2 network technology to achieve direct data transfers between two hosts on the same Ethernet network by bypassing the CPU or Kernel.
Currently RoCE Version 1 does not support Internet Protocol (IP) level routing and is not available in a multi-tenant domain. RoCE Version 2 does support IP level routing but is not available in a multi-tenant domain. There are currently no mechanisms to use RDMA data transfer protocols such as RoCE or iWARP using an overlay network that supports multi-tenancy in a Software Defined Data Center.
Today, the encapsulation of an overlay network L2 frame originating from a VM is handled in software. This involves significant work on the part of the overlay network software to maintain and lookup overlay network address mapping tables, and to encapsulate a transmit packet with an outer header that includes relevant overlay network, UDP, IP and Ethernet information before it is sent out on the wire. Similarly the reverse process of decapsulation of the outer header for a received packet is also the responsibility of the software. Each mapping table lookup and encapsulation/decapsulation of packet in software incurs significant CPU cost and affects performance.